Following methods are the best practices to design stable systems in complex environment. There are of course much more patterns but these are particularly the patterns I have used on my projects.
– Decoupling: Software should be built in independent modules. Microservices is a very good way of decoupling. This pattern makes it possible to evolve independent modules easily and fixing a problem in a module is much easier.
This pattern makes scalability also much easier.
– Timeout: A thread blocks until a resource become available, like a response from a remote call. This behaviour makes the systems unstable. To prevent this, a thread must to be able to time out. Building a timeout mechanism makes the systems more stable and reliable.
Using a timeout in Java calls or Oracle Service Bus callouts makes the system much stable and also responsive.
– Circuit Breakers: Using this patten (more info: https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern) makes the software more stable. Most application servers makes services available for this pattern and can be used in combination with the software implementation to make the systems more stable.
An example of circuit breaker implementation is “Test Connections On Reserve” checkbox of Weblogic server.
– Bulkheads: Use of Bulkheads pattern can be used to constrain the amount of concurrent requests and also prevents resource starvation. Use of thread pools or semaphores are implementations of this pattern. Throttling of services can also be seen as an implementation of this pattern, depending de implementation details. Hystrix framework of Netflix implements this pattern.
An example of this pattern is the Work Managers of Weblogic server.
– Limiting the load: Before invoking the resource, negotiating about the availability of the resources. Throttling can be seen as an implementation of this pattern.
An example of this method is Throttling of Business Services of Oracle Service Bus