Wednesday, November 23, 2016

The Power of Intimacy

A question I get asked a lot is the magic formula to do great work. There is an inherent charm in believing that an exotic potion is all one needs to surpass himself, make his mark in the world and live a fulfilling and complete life.

However, the wiser of us know that there is no secret elixir hidden somewhere in the woods. We all know the standard recipe for success, which includes a heavy dose of hard work, a large dollop of perseverance and a dash of madness.

Let us now focus on the components discussed above. Madness needs a daredevil attitude, a love for the unexpected and a high degree of confidence in self. However, the other two attributes can be a bit elusive especially when demanded over a long period of time, consistently.

The next big question is, then, how does one grab the bull by the horns? I commiserate with the reader if he has felt the pangs of frustration when trying to be persistent and keeping the work levels at the maximum. Maintaining hard work and perseverance can be frustrating, especially if the results are not immediate and tend to be over the horizon.

What is the key then? The key is to love your task so much that you forget all but your craft. The key is to feel so passionate about the task at hand that any distraction, any shiny short term glory seems devoid of sheen when put in contrast with the magnificent glory of your innate fire which burns only for the task.

The road to Rome is not so simple, though. How does one love a task so much? How does one ignite the fire of passion which will keep burning without being quenched by greed and glory? How does one truly feel the power of desire?

The key to that is a key to not only greatness in work, but greatness in all aspects of the life. That key will allow you to feel the love for many critical parts of your life, and when you put in hard work, perseverance and a ting of madness in those parts of your life, a legendary and happy life will greet you with open arms.

The way to love something or someone is to appreciate the beauty that lies inside the object, person or task. The way is to see what only you can see, to gaze in wonder at the magical components of the subject and to truly appreciate the chance to witness such a wonder.

Which brings me to the most important point of all. To witness such a beauty, to enter the inner sanctum, you need to know the subject to a deep level, and to know something or somebody to the core, you need to intimately engage with them.

Take a look at a lego box. The small insignificant pieces that are scattered around, when combined in just the right way, build an astonishing piece of marvel. It is the same with anything else. The small components, gestures and emotions which might be touted as insignificant and not deserving of your priceless attention might be the cogs in the machinery of magic that you wish to build.

When interacting with your life, be passionate and inherently curious. Know the inner details of your task. Know exactly how the radio that you are trying to fix works. Know the exact emotions that someone important in your life will be feeling when faced with a situation. Be intimate to a level where the object or person has no qualms in spilling their inner workings and feelings to you, so that you become a piece in the jigsaw puzzle and help in making the object or the life of the person important to you, better.

There is an enigmatic charm and joy in knowing that you connect with a piece of nature to the core and can contribute to the improvement of the same. When you focus all your attention and curiosity at something and someone, the outer layers just keep falling away and what is left is what matters.

Peace  

Monday, June 27, 2016

Resource Queues 2.0 -- The Next Version of Resource Management in Greenplum

Hi All,

This post covers my work around the next version of Resource Queues in Greenplum. Resource Queues are resource management for queries within Greenplum. This basically means that any query (except superuser queries) need to be run within limits of a given Resource Queue. This ensures that resource utilization is limited and can be monitored and modified. This also allows per role based resource usage prioritization.

Resource Queues allow the defining of three resource limitations. Number of queries that can be run by a role at a given point of time, the maximum amount of memory that can be consumed by all the queries running concurrently under a user at a given point of time, and the maximum cost that can be allowed for a plan selected in per role basis. So you can effectively limit the three parameters per role and that can be modified as you go ahead.

The first implementation of Resource Queues had some major issues around deadlocks. I will explain the problem further down.

Resource Queues use database locks inside Greenplum. What that implies is that the locks taken by Resource Queues is the same as being done by database shared objects, such as tables. Now, Greenplum acquires relation locks at various points in the query lifestage. For eg, in query execution. Resource Queue limits are evaluated after the planner stage, so that Resource Queue planning cost limits can be evaluated. If the resource limits are being exceeded by the current query, the current query will be made to sleep until another query releases the resources being used and then the current sleeping query will be woken up and the resource requirements will be checked.

Now, there are some cases where this steps on each other's toes. For example, consider the following case:

Query 1 starts
Query 2 starts

Query 1 is made to sleep due to excessive resource usage
Query 2 gets the Resource Queue slot lock

Query 1 is waiting for the Resource Queue lock.
Query 2 is waiting for a relation lock held be Query 1.

This leads to a deadlock.

This was a long standing problem.

The solution implemented in my new approach is as below:

Get the Resource Queue Slot lock prior to any other potential lock. This can be at the time of query admission control. This has the problem that the cost based checking cannot be done since that is a post planner stage task. For this case, the solution entails holding the lock till that stage but using a waiter queue to manage the resource queue slot locks. This basically means that if cost based checking exceeds the limit for a specific query, the query releases all the locks it holds and enters a waiter queue. Now, any other query waiting for the lock shall be ahead of the current query in the waiter queue, so will get the lock first. In the above case, Query 2 will hold Resource Queue Slot lock and relation lock so it can proceed.

Query 2 will then release its locks and Query 1 will get both Resource Queue lock and the relation lock so it can proceed.

This removes the non deterministic locking that happens around Resource Queue Slot locks and shared database object locks. The core reason is that relation locks are acquired sometimes during the parser stage as well because we do not want underlying relation to change as we proceed. This leads to a circular deadlock issue.

The pull request for the implementation is: https://github.com/greenplum-db/gpdb/pull/739

This serves as a good PoC to demonstrate resource management without deadlocks. Ideally, Resource Queues should not be using the same shared database object locks since they are essentially not the same but that is a different problem space.