time space compression and space time comprssion difference

What are these magical time-series compression algorithms?įirst of all, they’re not magic, but clever computer science techniques. Achieving 95% storage can save you over $10K-$25K per year in storage costs alone ($12K/10TB * 10TB/machine * 2 machines * 95% savings = $22.8K). A 10TB disk volume in the cloud is more than $12,000 per year itself ( at $0.10/GB/month for AWS EBS storage ), and additional HA replicas and backups can grow this number by another 2-3x. These algorithms also lead to compute performance improvements: as more data fits in less space, fewer disk pages need to be read to answer queries.

This translates into 90%+ storage cost savings, which can mean thousands of dollars (and in some cases, tens of thousands of dollars) of savings per year. According to our users, these algorithms have helped them achieve 90%+ lossless compression rates. In order to achieve high-performance while maintaining resource efficiency, we first identified several best-in-class time-series compression algorithms, and then implemented them in TimescaleDB. As a result, time-series datasets often scale well into the terabytes and more. Some time-series workloads also have high insert rates (e.g., IT monitoring, IoT sensor data). In order to analyze data over time, we insert new data (i.e., instead of updating existing data) on every measurement. One of the challenges of time-series data is storage footprint. Whether we are measuring our IT systems, web/mobile analytics, product usage, user behavior, sensor/device data, business revenue, etc., time-series data flows through our data pipelines and applications, and enables us to better understand our systems, applications, operations in real-time. Why compression matters for time-series data We use the term “we” throughout this article to represent the engineers who developed this capability: Josh Lockerman, Matvey Arye, Gayathri Ayyapan, Sven Klemm, and David Kohn. We also explain how we implement them in TimescaleDB, the first open-source relational database to use these time-series compression algorithms, and achieve 90%+ storage efficiencies. To do this, we explain how several lossless time-series compression algorithms work, and how you can apply them to your own projects. In this post, we set out to demystify compression. As a concept, compression is even older: “Morse code, invented in 1838, is the earliest instance of data compression in that the most common letters in the English language such as “e” and “t” are given shorter Morse codes.” ( source) Enter compression: “the process of encoding information using fewer bits than the original representation.” ( source)Ĭompression has played an important role in computing for several decades. The more efficiently we can represent that information, the more we can save on storage, compute, and bandwidth.

And as computing infrastructure has gotten cheaper and more powerful, we have asked it to represent more and more of our information, in the form of data we collect (which is often time-series data).īut computing is not free. Here’s how they work.Ĭomputing is based on a simple concept: the binary representation of information. Delta-delta encoding, Simple-8b, XOR-based compression, and more - These algorithms aren't magic, but combined they can save over 90% of storage costs and speed up queries.

Time space compression and space time comprssion difference