Skip to content

Fixed event counters (busy/idle connections count)#3476

Closed
neyromant wants to merge 4 commits intonpgsql:mainfrom
neyromant:fix_event_counters
Closed

Fixed event counters (busy/idle connections count)#3476
neyromant wants to merge 4 commits intonpgsql:mainfrom
neyromant:fix_event_counters

Conversation

@neyromant
Copy link
Contributor

If a non-canonical connection string is used (e.g. trailing semicolon), EventCounters produces idle and busy connections count different from the actual ones.

The reason:

  • PoolManager.Pools can contain duplicate pools under different keys (one corresponds to the original connection string, the other to the canonical one)
  • NpgsqlEventSource sums the connections count from all values ​​in the PoolManager.Pools

Copy link
Member

@roji roji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neyromant thanks for spotting this - you're right with your analysis of the bug.

The fix looks good overall (see some comments below). However, I'm a bit worried about adding more book-keeping inside PoolManager, specifically with respect to #3396. If we wanted to, we could also fix this by deduplicating the pools inside NpgsqlEventSource with a HashSet (though it's true this is better done once when adding pools, rather than every time).

@vonzshik what do you think?

PS I've opened #3480 to have an issue tracking this (not just a PR)


var idx = 0;
ConnectorPool? existsPool = null;
for (; idx < _pools.Length; idx++)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that in NpgsqlConnection we've already found the pool via the canonical/normalized connection string, so we could just call a different method here rather than searching for it again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I understood correctly ... Made an additional parameter to let me know if a new pool is being registered or an existing one.

@neyromant
Copy link
Contributor Author

neyromant commented Jan 26, 2021

@neyromant thanks for spotting this - you're right with your analysis of the bug.

The fix looks good overall (see some comments below). However, I'm a bit worried about adding more book-keeping inside PoolManager, specifically with respect to #3396. If we wanted to, we could also fix this by deduplicating the pools inside NpgsqlEventSource with a HashSet (though it's true this is better done once when adding pools, rather than every time).

@vonzshik what do you think?

PS I've opened #3480 to have an issue tracking this (not just a PR)

Thanks for the review, I made some edits.
I also like the idea with HashSet for pools in NpgsqlEventSource, but I probably have to wrap the addition in lock?

// NpgsqlEventSource:
// internal void PoolCreated() => Interlocked.Increment(ref _pools);
internal void PoolCreated (ConnectorPool pool)
{ 
  lock(Lock) 
   { 
     PoolsHashSet.Add(pool);
  }
} 

If lock is enough to solve the thread safety, I can do a different PR

@vonzshik
Copy link
Contributor

If we wanted to, we could also fix this by deduplicating the pools inside NpgsqlEventSource with a HashSet (though it's true this is better done once when adding pools, rather than every time).

I'll all for this, as the pool manager is already not that simple. As a possible solution, we can set a flag every time we add or remove a pool to reset the HashSet.

@roji
Copy link
Member

roji commented Jan 26, 2021

I think actual addition/removal of pools should be rare enough that a simple lock around the HashSet should be fine - even after #3396 - do you agree @vonzshik?

BTW I'd keep the HashSet on the PoolManager rather than on the EventSource - it could be useful for other purposes (e.g. tests).

@vonzshik
Copy link
Contributor

I think actual addition/removal of pools should be rare enough that a simple lock around the HashSet should be fine - even after #3396 - do you agree @vonzshik?

BTW I'd keep the HashSet on the PoolManager rather than on the EventSource - it could be useful for other purposes (e.g. tests).

I personally just don't like the idea of adding another array for the PoolManager to handle - for me it sounds like asking for more bugs. But if no one else has any objections, then I guess...

@roji
Copy link
Member

roji commented Jan 26, 2021

@vonzshik I'm OK either way. It should be OK on PoolManager - as long as we make it clear that the pools array/HashSet isn't reliable in the same sense as the one actually used when opening a connection. But am also OK with keeping it out.

@neyromant
Copy link
Contributor Author

@vonzshik @roji
I think that will not be able to use the HashSet directly - it is not thread safety. We could have used ConcurrentDictionary, but the fact is that getting Keys from it leads to increased memory traffic.
We rarely write to an array of pools, but we often read it in NpgEventSource.

I created a pr #3481 with an alternative approach - accounting for pools in NpgsqlEventSource.
There, as in the current review, there is no deduplication of pools as such. Everything is based on the correct logic in NpgsqlConnection.GetPoolAndSettings

@neyromant
Copy link
Contributor Author

Fixed in #3485

@neyromant neyromant closed this Jan 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants