Mgopurge 1.8 in candidate


#1

Mgopurge 1.8 has been officially pushed to the candidate snap channel. With this version the major change is to how we find what transactions can be pruned. The new algorithm starts removing transactions faster, and in practice is able to remove them much quicker overall.

To get the new release you can run: sudo snap refresh mgopurge --channel=candidate. It will likely be pushed to the stable channel early next week.

It exposes 3 new command line options and deprecates 1 old option:

  • multithreaded=true/false: The new algorithm defaults to running 2 parallel threads, one starting at the oldest transactions and one starting at the newest. This generally gets about a 50% improvement in transaction processing speed, at a cost of more load on the system.

  • prune-batch-size=INT: the new algorithm reads 1000 transactions at a time, and then finds the referenced documents. It cleans up the transaction queues of the referenced documents before pruning the transactions from db.txns. Changing the batch size changes how many transactions are considered in one pass. Setting it to 100 significantly slowed down pruning speed, and setting it to 10,000 did not significantly improve the speed.

  • prune-batch-sleep-ms=INT: Time-in-milliseconds to sleep between each batch. This allows you to reduce load on the system, allowing other processes to get queries in. Generally, processing a batch of 1000 txns takes about 100ms, so setting this to 10ms slows the process down by about 10%.

  • max-txns now deprecated. This referred to the old algorithm evaluating 1M transactions to analyze at one time. Since we are now able to work in much smaller batches, we no longer need to have a high upper cap.

The logic updates in the “prune” stage are likely to come to Juju 2.5.1.