Use std::unique_ptr instead of std::shared_ptr for inference manager (#5003)