Setup Serilog for Amazon OpenSearch – Asp.net

I’ve tried your setup and here are some results (note using only stable versions of software):

• .NET Core v6.0 (not a beta)
• Serilog.Sinks.Elasticsearch v 8.4.1
• Serilog.AspNetCore 5.0.0

Docker-compose used:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.0.1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.0.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

Program.cs

using Serilog;
using Serilog.Events;
using Serilog.Sinks.Elasticsearch;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddRazorPages();

builder.Logging.ClearProviders();

Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg));

ServicePointManager.ServerCertificateValidationCallback =
    (source, certificate, chain, sslPolicyErrors) => true;

var logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        AutoRegisterTemplate = true,
        MinimumLogEventLevel = LogEventLevel.Information,
        FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
        EmitEventFailure = EmitEventFailureHandling.RaiseCallback | EmitEventFailureHandling.ThrowException,
        TypeName =  "_doc",
        InlineFields = false

    })
    .CreateLogger();

builder.Logging.ClearProviders();
builder.Logging.AddSerilog(logger);

    

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}


app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

There are few things that should help with troubleshooting

1. Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg)); – will show real error from Serilog (most likely you will see SSL)

2. ServicePointManager.ServerCertificateValidationCallback = (source, certificate, chain, sslPolicyErrors) => true; – stop any SSL issues for now (you can fix them later)

3. The next thing I catch is issue with [_type] field that is generated by Serilog and is not accepted by Elastic > v8.2, this most likely will happen because your buffer keep old records.

4. While latest beta version of Serilog adopted to use TypeName="_doc", AWS opensearch 2.0.1 have another bug with "compatibility.override_main_response_version=true" setting (see details here)
   https://github.com/opensearch-project/OpenSearch/pull/3530 – basically I would suggest to rollback AWS OpenSearch to v2.

Afterwards it hopefully should work 🙂