How to Do Bulk Insert in ElasticSearch in Go? 🔎

Using ElasticSearch with Go, I encountered a specific challenge that consisted of sending an amount of information to an Index in ES. For this I didn't find clear documentation that resolved my question about how it worked and in my case why it failed.

Browsing on StackOverflow I found a solution with some errors and proceeded to optimize and fix it.

It consisted of creating a function that would serialize the information within the array and then save this already serialized information in a buffer to then send it.

Here's the solution in code with some specific cases due to the program I was making. You can change it and adapt it to your business case.

~

func ParseToNDJson(data []domain.ElasticLog, dst *bytes.Buffer) error {
enc := json.NewEncoder(dst)
for _, element := range data {
	if err := enc.Encode(element); err != nil {
		if err != io.EOF {
			return fmt.Errorf("failed to parse NDJSON: %v", err)
		}
		break
	}
}
return nil
}

func (edb *ElasticDB) SetElasticData(el []domain.ElasticLog) error {
var body bytes.Buffer
if err := ParseToNDJson(el, &body); err != nil {
	return fmt.Errorf("error encoding request: %s", err)
}
_, err := esapi.BulkRequest{
	Index: "logs",
	Body:  strings.NewReader(body.String()),
}.Do(context.Background(), edb.Client.Transport)
if err != nil {
	return err
}

return nil
}