X
tripwire
home demo order blog help faq
Data compression
posted 2017.11.08 by Clark Wilkins

The original design of TRIPWIRE assumed that each "parser" would report a different dataset. So we structured it where reporting the helium level for a Symphony was treated differently from reporting it for an Avanto. And it started to get silly when we started seeing Avantos and Esprees with differing Magnet Supervisory Units that we had to run specific parsers on, and the resulting being two Avantos being treated differently.

It was a tough problem to clean up though, and we kept "kicking the can down the road". Until today, when we ran into a bug affecting alerts on Espree helium levels (from v3.12). During the fix, I noted we were up to 160 different reported values — with significant overlap. Something had to be done.

v3.14 is an internal change that brings the reported values from 160 to 53. There are no more duplications, so helium level means the same things for all seven current data parsers. This patch took quite a bit of work to construct and then update > 16M records on the server, but it needed to happen at some point prior to the design work for TRIPWIRE 4, so today was the day.

As an illustration of the size of the patch, here's the execution log.


DELETE 342207
DELETE 485
DELETE 28646
DELETE 28645
DELETE 28645
DELETE 28645
DELETE 28645
DELETE 15442
DELETE 7673
UPDATE 0
UPDATE 0
UPDATE 25783
UPDATE 300875
UPDATE 15442
UPDATE 25783
UPDATE 300875
UPDATE 15442
UPDATE 25783
UPDATE 300875
UPDATE 15442
UPDATE 25783
UPDATE 28646
UPDATE 92662
UPDATE 33401
UPDATE 28645
UPDATE 110621
UPDATE 0
UPDATE 28017
UPDATE 28646
UPDATE 33401
UPDATE 25783
UPDATE 28646
UPDATE 33401
UPDATE 25783
UPDATE 95524
UPDATE 25783
UPDATE 28645
UPDATE 95524
UPDATE 25783
UPDATE 40337
UPDATE 33401
UPDATE 28645
UPDATE 33401
UPDATE 25783
UPDATE 28645
UPDATE 33401
UPDATE 25783
UPDATE 33401
UPDATE 25783
UPDATE 28645
UPDATE 28645
UPDATE 33401
UPDATE 15442
UPDATE 25783
UPDATE 28646
UPDATE 33401
UPDATE 25783
UPDATE 28646
UPDATE 25783
UPDATE 28645
UPDATE 25783
UPDATE 28645
UPDATE 25783
UPDATE 657704
UPDATE 28646
UPDATE 19993
UPDATE 28646
UPDATE 25783
UPDATE 28646
UPDATE 25783
UPDATE 28646
UPDATE 15442
UPDATE 25783
UPDATE 28646
UPDATE 15442
UPDATE 25783
UPDATE 28646
UPDATE 15442
UPDATE 25783
UPDATE 28645
UPDATE 25783
UPDATE 28645
UPDATE 25783
UPDATE 28645
UPDATE 25783
UPDATE 113483
UPDATE 15442
UPDATE 25783
UPDATE 113483
UPDATE 15442
UPDATE 25783
UPDATE 0
UPDATE 28646
UPDATE 19993
UPDATE 28645
UPDATE 28646
UPDATE 19993
UPDATE 33401
UPDATE 33401
UPDATE 33401
UPDATE 33401
UPDATE 25783
UPDATE 28645
UPDATE 19993
UPDATE 28645
UPDATE 19993
UPDATE 28645
UPDATE 19993

We also turned off the old parsers for good today — fully commiting to the v3.12 changes.

© 2014-2017, simplexable, llc
history