tomcat - Connection Pool Empty Hibernate 4, but Unable to Find the Culprit -


i'm monitoring sql database connections every 5 minutes. days it'll hover around 5 connections (my idle) i'm @ 50. recursive issue because can't see why jump 5 50 within 5 minutes 0 traffic.

i'm using hibernate 4 , tomcat , know of issue in hibernate patched in 4.3.2, i'm on 4.3.5

more details: pool empty event happens every day @ 7:13:20pm... sounds automatic. using quartz , runs every 1 minute, can't see how they're related.

my properties:

jmxenabled = true initialsize = 5 maxactive = 50 minidle = 5 maxidle = 25 maxwait = 10000 maxage = 10 * 60000 timebetweenevictionrunsmillis = 5000 minevictableidletimemillis = 60000 validationquery = "select 1" validationquerytimeout = 3 validationinterval = 15000 testonborrow = true testwhileidle = true testonreturn = false jdbcinterceptors = "connectionstate" defaulttransactionisolation = java.sql.connection.transaction_read_committed

environment:

  • tomcat 7.0.59
  • java 1.7.0 update 76
  • sql server 2012

more information: reduced quartz job frequency every 5 minutes. event still took place when loaded page/view in application. @ 7:14 pm. i'm on verge of downgrading hibernate 3.

update today reloaded application in tomcat manager @ 6:50 pm, event still took place. thread dump

ah kind of bugs fun. there no way can point exact culprint (except when digs bug in libs mentioned), lets see how can debug this. approximatly easy hard do, although details depend on environment.

  1. you have helpful information: problem happens @ same time. hints @ 2 options: either 1 of jobs run quartz eats connections, or (possible external) happening @ time causing code eat connections. should check job configurations , cron jobs or jobs configured inside database or similar potential culprits. note might start quite time earlier , arrive @ critical state later on, job might start 2hours earlier know.

  2. check logs , system logs , database logs happens @ time or time earlier.

  3. double check gets connection if returns connection. when exceptions thrown. 1 classical way fail @ construction (java pseudo code):

    connection con;  try {     con = getconnection();     statement = stmnt = con.createstatement();     .... } (exception ex){     if (stmnt != null) stmnt.close();     if (con != null) con.close(); // never happen if stmnt.close throws exceptions  }

  4. establish logging let see when connection don't returned. start in application should go through kind of wrapper (aop around aspect, servlet filter or similar). wrapper should following: create unique id action (uuid) , put in mdc of logging framework. @ end of action id gets removed again. other logging should include id. wrap connection pool well. keep track on when requested connection, including timestamp, id , possibly stacktrace (by creating , storing exception). log that. every time connection gets returned log time used. every time connection gets requested check if connection used longer threshold.

  5. isolate things: set second server, run application. have same problem? run parts on 1 of 2 servers, still both have problem? continue exclude candidates until 1 left.


-

Comments

Post a Comment

Popular posts from this blog

apache - PHP Soap issue while content length is larger -

we have developed soap based webservice post data third party webservice. accepts 4 params. in 1 of parameters accepts xml string no size limit restriction put in place. the webservice works fine in local environment running on php 5.2 , apache2. works content length long 10 mb. in staging server running nginx , php fpm.we have updated max post size , max upload filesize in php.ini , client_max_body param in nginx.conf allow upto 50 mb. but webservice client keeps on loading in staging environment. gives 502 timeout , gives soap error. can me understand issue here?
Read more

asynchronous - Python asyncio task got bad yield -

i confused how play around asyncio module in python 3.4. have searching api search engine, , want each search request run either parallel, or asynchronously, don't have wait 1 search finish start another. here high-level searching api build objects raw search results. search engine using kind of asyncio mechanism, won't bother that. # no asyncio module used here class search(object): ... self.s = some_search_engine() ... def searching(self, *args, **kwargs): ret = {} # raw searching according args , kwargs , build wrapped results ... return ret to try async requests, wrote following test case test how can interact stuff asyncio module. # here testing script @asyncio.coroutine def handle(f, *args, **kwargs): r = yield f(*args, **kwargs) return r s = search() loop = asyncio.get_event_loop() loop.run_until_complete(handle(s.searching, arg1, arg2, ...)) loop.close() by running pytest, return runtimeerror: task got bad yield : {results s...
Read more

javascript - Complete OpenIDConnect auth when requesting via Ajax -

the normal openidconnect server works like: you go a.com/secure-resource you 302 server your browser handles , sends identity server you login there it sends a.com via post you logged in on a.com , a.com/secure-resource on browser. however have scenario i'm trying solve need help. the user logged in on idserver the user logged in on a.com the user not logged in on b.com we need send ajax call web server b.com (from domain a.com ) b.com configured use openidconnect. but because request b.com via ajax, user cannot redirected idserver. (all in response 302 ) we can go ahead , handle 302 via ajax (i'm still not sure whether work, security-wise). but is there scenario in identityserver/openidconnect designed these situations? with identityserver in scenario setup server b.com use bearer token authentication, need use access token provided a.com in headers of ajax call $.ajax({ url: 'http://b.com', headers: { ...
Read more