DPA在电信计费系统故障诊断案例

3月20日中午,电信客户的计费系统故障,所有操作响应缓慢,应用无法正常运行。现场DBA已经迅速将应用连接转移到RAC集群的另一节点,恢复应用服务,我们使用DPA来回看事故现场,此时离故障发生,已经一小时了……

回溯负载情况,如下图

 

问题最严重时,实例堵塞了近700个连接操作…针对此时间段,进一步点击、下钻分析当时系统发生了什么

系统先是每分钟4~5次堵塞(像是咳嗽),到了13:40后,就处于长时间“梗”住的状态,对于外部业务来说,数据库实例已不可用了。

 

利用DPA,继续下钻,查看堵塞原因:

1654#会话导致80个会话被堵塞,这80个会话的等待事件是 sga:allocation forcing component growth,这80个会话中的一些会话又堵塞了一些其他会话,如下图

266号会话堵塞了3个会话,这三个会话的等待事件都是cursor:pin s wait on x,这个等待事件一般都是跟sql parse相关,所以,现场工程师开始查看oracle 自己的ADDM报告,说是解析有问题,建议应用分析解决问题…被误导了

 

至此,DPA下钻后清晰显示,堵塞的根源是1654号会话,DPA显示是MMAN进程,它堵塞了80个会话,这些会话又堵塞了更多其他会话,是一个3级的堵塞关系,幸好DPA以“树型”显示方式,把这种多层级堵塞关系显示的非常清晰。

结论:Oracle MMAN进程进行过于频繁的SGA内存组件(本例中是share pool)resize操作,导致了问题…

 

这张图中显示的share pool内部情况,看官能看出异常么?

 

“划一条线1美元,知道在哪里划线9999美元”。

至此,后续我们如何解决这个问题的就不再讨论了,解决问题从来不是难事,难的是发现问题。

后记: 从DPA的负载雷达上,我们看到在系统出现问题之前的一小段时间内,系统负载不正常的降的很低,其后负载飙升上去…… 那段时间负载异常降低,说明了什么? 应用程序发生了什么…… 这些从Database Performance Analyzing层面,就无从知晓了。

96 Comments

  1. Pingback: mens nike jordan air spike 40 forty pe fire red white fire red black

  2. Pingback: DeVondre Campbell Road Jersey Nike 59 Atlanta Falcons Vapor NFL Atlanta Falcons Limited Red Home Nike Jersey

  3. Pingback: 15 inch computer bag pinterest.com

  4. Pingback: nike 5.0 running shoes black

  5. Pingback: nba shoes lebron

  6. Pingback: nike air force 1 low 07 lv8 mini swoosh barcelona uk trainers

  7. Pingback: adidas rose collection

  8. Pingback: tshirt champion tshirt

  9. Pingback: dunk sky hi glitter

  10. Pingback: jewelry findings

  11. Pingback: asics shoes gt 2000 pinterest.com

  12. Pingback: 褋械褉械斜褉褟薪褘泄 芯褉邪薪卸械胁褘泄 asics tigers shoes

  13. Pingback: uk trainers womens converse cowboy cloth triple black

  14. Pingback: air jordan iv red laser

  15. Pingback: marine schwarz converse embroidery young

  16. Pingback: nike air 110 twitter.com

  17. Pingback: uk trainers jordan 11 green snake black sonic volt ice yellow

  18. Pingback: nike tiempox finale tf black white metallic vivid

  19. Pingback: nike sock dart breathe 819686 200

  20. Pingback: uk trainers nike fs lite run sienna

  21. Pingback: ayunsue couple winter duck down jacket men 2018 new canada outdoor racoon fur collar camouflage long coat women

  22. Pingback: p煤rpura gris adidas consortium ozweego 2

  23. Pingback: air jordan 29 wing

  24. Pingback: undefeated assc x undefeated size us xl eu 56

  25. Pingback: annex ventilator twitter.com

  26. Pingback: kenzo jogging shorts

  27. Pingback: nice wedding shoes for bride wanelo.co

  28. Pingback: nike air max 2011 men white black shoes2017 nike air max sale

  29. Pingback: nike metcon 4

  30. Pingback: 褉芯蟹芯胁褘泄 蟹械谢械薪褘泄 nike hyperdunk 2016 low

  31. Pingback: 泻褉邪褋薪褘泄 芯褉邪薪卸械胁褘泄 adidas crazy

  32. Pingback: nike soccer cleats 2018 nike hypervenom phantom iii df fg

  33. Pingback: nike dri fit hustle knit mens basketball jacket3

  34. Pingback: adidas crazylight boost 2016 low bb8175

  35. Pingback: 蟹芯谢芯褌芯 卸械谢褌褘泄 nike air pippen 1

  36. Pingback: mens nike air max 2012 white and dark grey running shoes

  37. Pingback: deep v neck sleeveless dress

  38. Pingback: mens big tall long sleeve hooded fleece sweatshirt goodfellow co target

  39. Pingback: nike elite black photo blue metallic silver action green performance big kids boys basketbal

  40. Pingback: reebok instapump fury bs63670

  41. Pingback: supreme camo t shirt

  42. Pingback: the north face mens aconcagua vest asphalt grey s

  43. Pingback: white converse with stars

  44. Pingback: d lillard 2 black and blue twitter.com

  45. Pingback: urban outfitters champion beanieall weekend aspen

  46. Pingback: cinderella costume adult twitter.com

  47. Pingback: chic multi colors wedding dresses 2018 ball gown crystal off the

  48. Pingback: bustier adult costume nightdress transparent dress bikini dress

  49. Pingback: fs men s timberland boots size 46 12

  50. Pingback: hq kansas air national guard military ball cap

  51. Pingback: havaianas women`s flip flops slim mermaid sandal crocus rose swarovski crystal

  52. Pingback: adidas originals tubular entrap w s75915

  53. Pingback: nike air monarch iv lifestylegym shoe

  54. Pingback: new era ny hat greyblack

  55. Pingback: create your own custom jerseys football jerseys thesporting

  56. Pingback: good cheap sports bras

  57. Pingback: farm house key hanger w 4 hooks personalized

  58. Pingback: huarache black

  59. Pingback: womens melissa vivienne westwood black

  60. Pingback: awesome 2015 2016 footballshirt footballshirts home jersey juventus

  61. Pingback: hair extensions black 27inch

  62. Pingback: oliver red label luchtige wollen beanie

  63. Pingback: air jordan 1 explorer xx size 9

  64. Pingback: the lion king phone cases cover for samsung galaxy 2015 2016 j1 j2 j3 j5 j7

  65. Pingback: 2016 stylish korean version baseball cap advertising hats men and cap team cap

  66. Pingback: 泻芯褉懈褔薪械胁褘泄 蟹芯谢芯褌芯 d rose 6

  67. Pingback: supreme x the north face 16ss steep tech t shirt white

  68. Pingback: beige voltio converse two layer canvas

  69. Pingback: under armour men s ua antler logo adjustable cap black dynamite one size fits

  70. Pingback: 80 s wig men rockstar black slash kiss rocker costume wigs fits women kids

  71. Pingback: nike beanies saleperfect nike beanie black cool grey black black for

  72. Pingback: 803174 413 nike hyperdunk 2015 low limited city pack rio men women

  73. Pingback: product image nike mens zoom rival md 8 track and field shoes

  74. Pingback: womens mizuno wave rider 10.5

  75. Pingback: womens air jordan 5 olympic

  76. Pingback: season 6 everything you need to know about the

  77. Pingback: womens red glitter eden shoes

  78. Pingback: new jordan 11 xi air max 09 sole fusion space jams black vars

  79. Pingback: adidas gazelle bb5476 unisex trainers in black white

  80. Pingback: men grey blue black air jordans

  81. Pingback: t shirt to match retro air jordan 8 championship shoe

  82. Pingback: sq732541 nike air zoom prestige tm yi4fqsx tennis fr nike chaussure homme da840189

  83. Pingback: nike air max ltd summer 2012

  84. Pingback: 2016 nike sideline vapor visor black

  85. Pingback: samsung series phone case exquisite pattern ultra thin anti scratch cover samsung s9 plus brown

  86. Pingback: womens gym training shoes 34

  87. Pingback: nike blue nike huarache

  88. Pingback: adidas kids neo bb9tis high top sneaker toddler shoes black blue cross

  89. Pingback: air max website

  90. Pingback: silver and diamond ring

  91. Pingback: womens dunk high nike shoes

  92. Pingback: air shoes nike lab red ultra aw max bw swvxdz

  93. Pingback: los angeles lakers mitchell ness back to back champions grey t shirt s

  94. Pingback: palace tri ferg pyramids iced out skate t shirt heather grey

  95. Pingback: address of mizzonis pizza ballyfermot mizzonis pizza ballyfermot dublin location zomato ireland

  96. Pingback: womens teething toys for german shepherd puppies

Leave Comment