> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Debugging Tensorfuse Deployments and Jobs with logs

> Debug your applications by querying past logs

When you deploy your applications using [`tensorkube deploy`](/reference/cli_reference/tensorkube_deploy) or submit your jobs using
[`tensorkube job queue`](/reference/cli_reference/tensorkube_job), Tensorfuse automatically collects logs from your application and stores them in a
centralised location configured while setting up Tensorfuse.

By default, Tensorfuse stores logs in AWS Cloudwatch so that they are easily accessible and can be queried using the AWS CLI or the AWS Console. This guide will
walk you through the process of querying your logs and will also equip you with a library of queries that can be run quickly to debug common issues.

## Pre-requisites

All your logs are accessible on the [Cloudwatch console](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:logs-insights).
Make sure that you have the following settings configured:

1. Under `Select log groups by`: `Log group name`
2. Under `Selection Criteria`: `/aws/containerinsights/tensorkube/application`

## Querying application logs

To focus on your application's logs, you can use [CloudWatch Logs Insights](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:logs-insights)
with efficient queries targeting your application's log streams. To view the logs, copy one of the queries from below
and run it on the [Cloudwatch console](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:logs-insights). Below are some useful queries:

### Filter logs by application name

```sql theme={null}
fields @timestamp as Time, coalesce(log_processed.message, log, @message.log) as LogMessage, kubernetes.pod_name as PodName
| filter @logStream like /YOUR_APPLICATION_NAME/ and kubernetes.namespace_name = "YOUR_ENVIRONMENT_NAME"
| sort Time desc
| limit 100
```

* Replace `YOUR-APPLICATION-NAME` with the name of your deployed application. You can get the application name using [`tensorkube deployment list`](/reference/cli_reference/tensorkube_deployment).
* Replace `YOUR_ENVIRONMENT_NAME` with the name of your [Tensorfuse environment](/concepts/environments) where the application is deployed. Tensorfuse uses `default` environment if no environment is mentioned
* The query retrieves logs for a specific application, sorted by timestamp.

### Search for error logs

Use this query to find potential issues in your application logs by filtering for error-related keywords.

```sql theme={null}
fields @timestamp as Time, coalesce(log_processed.message, log, @message.log) as LogMessage, kubernetes.pod_name as PodName
| filter @logStream like /YOUR_APPLICATION_NAME/ and kubernetes.namespace_name = "YOUR_ENVIRONMENT_NAME" and @message like /error|exception|fail/
| sort @timestamp desc
| limit 100
```

## Querying Job Logs

To focus on your queued jobs' logs, run the following query on the [Cloudwatch console](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:logs-insights)

```sql theme={null}
fields @timestamp as Time, coalesce(log_processed.message, log, @message.log) as LogMessage
| filter @logStream like /inference-job/
| sort @timestamp desc
| limit 100
```

This query retrieves logs for all queued jobs, sorted by timestamp.

You can further modify the queries by using the [CloudWatch Logs Insights Query Syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) to filter logs based on specific criteria.

<Warning>
  **Important**
  To avoid incurring excessive charges by running large queries, keep in mind the following best practices:

  * Select only the necessary log groups for each query.

  * Always specify the narrowest possible time range for your queries.

  * When you use the console to run queries, cancel all your queries before you close the CloudWatch Logs Insights console page. Otherwise, queries continue to run until completion.

  * When you add a CloudWatch Logs Insights widget to a dashboard, ensure that the dashboard is not refreshing at a high frequency, because each refresh starts a new query.
</Warning>