Operation and Custom Plugins for High Availability

Operation and Custom Plugins for High Availability Yahoo Japan Corporation Jun Cui January 7th, 2020

About me Jun Cui • Site reliability engineer at Yahoo! Japan(September 2017~) • API Gateway developer Hobby • Playing Games(PS4, Nintendo Switch) • Playing Monster Hunter World-Iceborne recently image: screenshot of CAPCOM CO., LTD. monster hunter world-iceborne Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 2

Agenda 1. Kong usage states in Yahoo! Japan 2. Kong system configuration in Yahoo! Japan 3. Deploy Kong with Ansible 4. Custom plugins 5. Future work Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 3

1.Kong usage states in Yahoo! Japan Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 4

About Yahoo! JAPAN 100+ services 150,000+ servers Reference: https://promotionalads.yahoo.co.jp/online/yj_mediasheet201812.html Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 74,500,000,000+ PV/month (average in 2018) image: aflo 5

Kong usage states ・Release timetable Kong EE 0.33-1 2018.10.1 Kong EE 0.33-2 Kong EE 0.34 2019.3 2019.5 Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. NEW! Kong EE 0.36-1 2019.9 6

Kong usage states ・ Configurations on Kong ・220+ Workspaces ・590+ Services ・690+ Routes ・ Req/sec ・7000+ average per Kong cluster ・12000+ peak per Kong cluster Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 7

2. Kong system configuration in Yahoo! Japan Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 8

Kong for Backend internal client Kong (east) Backend (east) Kong (west) Backend (west) GSLB ・2 Kong clusters for backend developers. ・GSLB (Global Server Load Balancer) is a network equipment which directs network traffic to a group of data centers in various geographical locations. ・Developers could choose one or two Kong clusters depending on their availability requirements. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.

Kong for Frontend external client Kong (east) Frontend (east) Kong (west) Frontend (west) GSLB ・2 Kong clusters for frontend developers. ・It proxies external requests to internal frontend servers. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.

Kong clusters Kong (east) external client Frontend (east) GSLB Kong (east) Backend (east) Kong (west) Backend (west) GSLB Kong (west) Frontend (west) ・4 Kong clusters in production environment. ・2 clusters for frontend and 2 clusters for backend. ・Total 240 Kong nodes in 4 clusters. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.

Active/standby in Kong clusters Kong (east) external client Frontend (east) GSLB Kong (east) Backend (east) Kong (west) Backend (west) GSLB Kong (west) Frontend (west) ・Kong checks upstreamʼs health on a regular basis. ・If one of FE/BE is not able to function normally, active/standby plugin failovers all the requests to the other FE/BE. ・After failed FE/BE has been recovered, proxies will be returned to the original FE/BE. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.

3.Deploy Kong with Ansible Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 13

Deploy Kong with Ansible Screwdriver.cd Artifactory yum publish commit YJ network Production environment git ansible-playbook ansible resource ssh ansible deploy server Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. Kong server Kong server Kong server

Package version and config file management group_vars/all/all.yml # kong.conf kong_log_level: error kong_proxy_listen_port: 80 # package version kong_custom_plugin: kong-plugin-active-standby-1.x.x, kong-plugin-sorry-page-2.x.x, kong-plugin-custom-1.0.0 Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.

4.Custom plugins Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 16

#1 Traffic abuse prevention Suspected to be abuse API node1 Kong Server API node2 API node3 request deny send event Traffic abuse prevention server ・Check accesses from Yahoo ID/remote IP/other attribute in fixed period. ・Deny requests when meet the limit, which configured by developers. ・Protect from DoS attack to reduce server load. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 17

#2 Sorry page API node1 Kong Server API node2 API node3 5xx return customize html/json ・Kong would return 503 if all targets of upstream become unhealthy states. ・Sorry page could respond to users with customized content and status code. ・Developers could customize content/status for each endpoint. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 18

#3 Active/standby API node1 Kong server upstream1 cluster1 API node2 API node3 route service API node4 active/standby plugin upstream2 cluster2 API node5 API node6 ・Proxy all requests to API node of cluster1 in normal. ・Switch all requests to cluster2 if cluster1 crashed. ・Enable to offer our services as a whole continually. ・Simpler implementation of active/standby than canary release plugin. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 19

Active/standby vs Canary ・ Canary ・ Active/standby "config": { "standby_host” } "config": { "duration", "groups", "hash", "percentage", "start", "steps”, "upstream_fallback", "upstream_host", "upstream_port", "upstream_uri" } Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 20

5. Future work Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 21

Future work ・Current Issues ・Users cannot see metrics data of their own service. ・Frontend developers do not expect to display Kong error to end-users. ・Deployment is a big work. ・Future work ・Provide Grafana to developers, using Kong vitals. ・Develop custom plugin to show customized error page instead of Kong error. ・Implement automatic deployment to reduce maintenance cost. ・Increase Kong nodes. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 22

Thank you !! Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 23