[Systems] (forw) [mirrorbrain] important: breakage in 2.11.0 on Debian/Ubuntu

[Peter Pöml]

Hi Bernie,

Am 03.12.2009 um 19:20 schrieb Bernie Innocenti:

> On Thu, 2009-12-03 at 13:34 +0100, Peter Poeml wrote:
>
>> I pushed out a new release with the fix (2.11.1), and fixed packages
>> are already available.
>
> Ok, picked them up.
>
>
>> I sincerely hope this didn't cause you inconvenience, because you had
>> not updatd yet...! I apologize for the error.
>
> No, luckily we had not upgraded yet. Thanks for notifying us.

Fine! Relieving to know...

> By the way, is it safe to switch to MySQL? Running two different RDBMS
> on the same server seems like a big waste of resources, and we already
> need MySQL for all the web applications.


MySQL doesn't support the database scheme that's used with PostgreSQL.  
Due to the usage of an array type, the database scheme can be  
unusually compact, and optimized for the task at the same time. In  
fact, the tools provided don't support MySQL anymore. mod_mirrorbrain,  
as the core, would probably still work, but it's not currently tested.  
mod_asn is strictly PostgreSQL-only, due to another special data type.

The resources that the additional server needs should be very, very  
small. It's RAM usage should not be more than what MySQL would need as  
additional memory, if it would serve the data; probably less.  
PostgreSQL is very lean, if it is not configured to use lots of  
resources.

I know, one usually fears that a database server will generally be a  
big thing, but PostgreSQL is only a few MBs. On a box that I rent, I  
run it with 64MB of buffer cache, the whole system has 1G, and I even  
had the entire openSUSE database on there: a million of files on 150  
mirrors... of course, the buffer cache was too small for that - it  
needs about a Gig to be performant. Maybe your concern is stronger  
than needed.

But how large is your data set?

You can find out by using 'psql' with the query shown below:

postgres at mirrordb:~> psql -U mb mb_opensuse
Welcome to psql 8.3.7, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

mb_opensuse=> SELECT relname, relkind, relfilenode, reltuples,  
relpages, relpages*8/1024 AS relMB FROM pg_class WHERE relkind IN  
('r', 'i') AND relname ~ '^.*(file|server|pfx|temp1).*' ORDER BY 1;
                 relname                 | relkind | relfilenode |  
reltuples  | relpages | relmb
-----------------------------------------+---------+------------- 
+------------+----------+-------
filearr                                 | r       |       17768 |  
1.8912e+06 |    54503 |   425
filearr_path_key                        | i       |       17777 |  
1.8912e+06 |    81031 |   633
filearr_pkey                            | i       |       17775 |  
1.8912e+06 |    38586 |   301
pfx2asn                                 | r       |       16813 |      
299034 |     3329 |    26
pfx2asn_pfx_key                         | i       |       16818 |      
299034 |     6906 |    53
pfx2asn_pkey                            | i       |       16816 |      
299034 |     2764 |    21
server                                  | r       |       16400  
|        189 |      156 |     1
server_enabled_status_baseurl_score_key | i       |       16412  
|        189 |        8 |     0
server_identifier_key                   | i       |       16410  
|        189 |        6 |     0
server_pkey                             | i       |       16408  
|        189 |        6 |     0
sql_sizing_profiles                     | r       |       11459  
|          0 |        0 |     0
temp1                                   | r       |     6831980 |      
764924 |     3385 |    26
temp1                                   | r       |     6863332  
|       2730 |       13 |     0
temp1_key                               | i       |     6831983 |      
764924 |     2100 |    16
temp1_key                               | i       |     6863335  
|       2730 |       10 |     0
(15 rows)

mb_opensuse=>



The rightmost column will show the table size in MB. reltuples is the  
number of rows in that table.


If you could look up those numbers, I could give some advise how much  
resources the database server should need.

(And I should finally write up all I know about database tuning for  
MirrorBrain in the documentation, including the above...)

By the way, if there is quite some turnover in the file system you  
might want to run "mb vacuum" every once in the while. For instance,  
weekly. This will remove all file entries from the database that are  
not (or no longer) on any mirror; dead rows, so to speak. It's not to  
confuse with PostgreSQL's vacuuming, which usually happens  
automatically nowadays (8.3 and onwards I believe).

The "mb vacuum" command also outputs some numbers, that would be  
interesting to see.

Thanks!
Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3914 bytes
Desc: not available
Url : http://lists.sugarlabs.org/private/systems/attachments/20091204/61e930cc/attachment.bin